tar 2
$TAR^2$: Temporal-Agent Reward Redistribution for Optimal Policy Preservation in Multi-Agent Reinforcement Learning
Kapoor, Aditya, Tessera, Kale-ab, Baranwal, Mayank, Khadilkar, Harshad, Albrecht, Stefano, Sun, Mingfei
In cooperative multi-agent reinforcement learning (MARL), learning effective policies is challenging when global rewards are sparse and delayed. This difficulty arises from the need to assign credit across both agents and time steps, a problem that existing methods often fail to address in episodic, long-horizon tasks. We propose Temporal-Agent Reward Redistribution $TAR^2$, a novel approach that decomposes sparse global rewards into agent-specific, time-step-specific components, thereby providing more frequent and accurate feedback for policy learning. Theoretically, we show that $TAR^2$ (i) aligns with potential-based reward shaping, preserving the same optimal policies as the original environment, and (ii) maintains policy gradient update directions identical to those under the original sparse reward, ensuring unbiased credit signals. Empirical results on two challenging benchmarks, SMACLite and Google Research Football, demonstrate that $TAR^2$ significantly stabilizes and accelerates convergence, outperforming strong baselines like AREL and STAS in both learning speed and final performance. These findings establish $TAR^2$ as a principled and practical solution for agent-temporal credit assignment in sparse-reward multi-agent systems.
- Information Technology > Artificial Intelligence > Machine Learning > Reinforcement Learning (1.00)
- Information Technology > Artificial Intelligence > Representation & Reasoning > Agents > Agent Societies (0.66)
- Information Technology > Artificial Intelligence > Machine Learning > Learning Graphical Models > Undirected Networks > Markov Models (0.46)
TAR 1.0 or TAR 2.0: Which method is best for you?
In Casepoint, for example, a user can begin a TAR 2.0 session by reviewing as few as 50 documents (although our recommended ranking threshold is every 100 documents), and at each ranking threshold, the model re-ranks the corpus automatically. Doing this in tandem with Casepoint's Dynamic Batching feature, the user ensures that they are always looking at the highest-ranked documents. This allows you to strengthen your model faster because TAR 2.0 will continue to present documents in the batches until none of the documents presented are of relevance. Another benefit of TAR 2.0 is the ability to run multiple sessions simultaneously, where each session represents a different legal topic or issue you are trying to find relevant documents for. Being able to "bucket" groups of documents by relevant issues and have people dive into the review right away is a huge step forward.